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| PREFACE 


Educational Testing Service (ETS) is a private, nonprofit, 
educational organization with primary involvement in the 
areas of measurement and research. In collaboration with 
a wide variety of institutions and agencies, ETS provides 
programs, research and services for the identification 
and recognition of individual talents and seeks to 
contribute to the expansion of opportunities available 

to all individuals. 


ETS recognizes its responsibility to the individuals and 
institutions that it serves directly and to the larger 
society that ultimately must judge the value and efficacy 
of its work. This responsibility is implicit in the 
educational character under which ETS operates and is 
reflected in the products and services offered by the 
organization since its inception in 1947. Periodic and 
intense self-examination has been one means by which ETS 
has attempted to assure its continuing commitment to the 
broad constituency which it serves. As a step in that 
continuing process, ETS has developed these Principles, 
Policies and Procedural Guidelines Regarding ETS Products 
and Services (Guidelines), first published on August 1, 
1977. 


The Guidelines are designed to ensure that ETS products 
conform to uniformly high standards with respect to seven 
areas of basic importance: Accountability, Confidentiality 
of Data, Product Accuracy and Timeliness, Research and 
Development, Tests and Measurement, Test Use, and Technical 
Assistance, Advice and Instruction. The first three 
sections of the Guidelines deal with issues that relate to 

| all ETS activities: Accountability, the responsibilities 
of ETS to those affected by its activities; Confidentiality 
of Data, the rights to and limitations on access to data 
collected by ETS; and Product Accuracy and Timeliness, the 
control of quality and performance according to scheduled 
commitments. 


The remaining sections concern issues specifically relating 
to ETS' main endeavors: Research and Development, Tests and 
Measurement, Test Use, and Technical Assistance, Advice and 
Instruction. 
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The Guidelines attempt to codify standards used in various 

ETS programs and services that should be considered for more 
general application and more formal articulation. ETS has 

not developed these Guidelines because of any legal 
requirements. They were not devised in response to any 
standards published by professional organizations, although 
ETS endorses the goals served by those efforts. These 
Guidelines are the result of an extensive deliberative process, 
which began at my direction in 1974. More than 150 members of 
the staff have participated in the initial development of the 
Guidelines and their subsequent review and revision. The 
Guidelines are drawn from the particular circumstances and 
needs at ETS and are designed to reflect its institutional 
objectives. Because of their origin and purposes, the 
Guidelines cannot generally or usefully be applied to 
organizations whose practices, programs or services differ 
from those of ETS. 


The Guidelines include three types of statements, which 
have varying degrees of generality and significance. The 
principle that should underline ETS efforts in any given 
area is set first; policies to govern decision-making, and 
designed to foster more specific goals, are set forth next; 
and procedures relative to the conduct of specific ETS 
activities are last described. However, because the 
guidelines must apply to a broad diversity of programs and 
services with differing characteristics, each procedural 
guideline represents only one method of achieving the 
objectives stated in the policies. 


Even the principles and policy statements might reasonably 

be expected to change in focus or coverage, as scientific 
developments or practical experience change the nature 

and desirability of various goals. Thus, the Guidelines 

are intended to encourage and not deter change and 
improvement. They should foster the development, exploration 
and use of alternative approaches that hold to high standards 
and serve important ETS policies. 


ETS believes that the Guidelines can contribute significantly 
to the quality and utility of ETS products and services. The 
Guidelines have been distributed to executive, managerial and 
professional staff at ETS so that they may be applied in the 
course of their work at ETS. ETS does not have complete 
responsibility or authority, of course, to determine how the 
Guidelines will be implemented in ETS programs for which 
policy is substantially established by a sponsoring group 
other than ETS. ETS has, however, taken steps to encourage 
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and assist those groups to implement the Guidelines as 
their activities relate to ETS. New activities ETS enters 
into will be held to this same expectation. 


During the first year after their introduction, the Guidelines 
existed in provisional form. They were used in evaluating 
all ETS programs and services, and, where a substantial Lack 
of compliance was found, steps were taken to achieve compliance 
within a reasonable time, or to revise particular provisions 
to reflect more accurately the diversity and practical demands 
of ETS operations. This process of review will continue as part 
of our organization-wide quality assurance effort and a report 
will be given to the Board of Trustees at regular intervals. We 
are committed to the effort to apply a set of positive criteria 
to our work and, taking the present document as a starting point, 
‘ we will go forward with the continuing process of improving 
those criteria and consequently the programs and services ETS 
provides. 


February 1, 1979 


(7-6 Saas 


William W. Turnbull 


President 
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ACCOUNTABILITY 


Principle 


ETS acknowledges responsibility for the 
effective stewardship of its resources 

to the New York Board of Regents which 

has issued its corporate charter; to the 
governing boards that sponsor and set policy 
for programs or services in which ETS 
products or services are used; to the 
individuals and committees that advise ETS 
with respect to appropriate policy for its 
programs; to the institutions and agencies 
that use ETS products and services; to 
persons who take ETS tests (and parents or 
guardians of minor persons), submit data 
for use by ETS or for distribution to 
others, or participate in research and 
development projects conducted by ETS; and 
to the professional associations that 

are concerned with educational and psycho- 
logical measurement and research. 


Policies 


A. ETS will furnish appropriate information to those to whom 
it is responsible so they may make informed, independent 
judgments as to the effectiveness with which ETS exercises 


its stewardship. 


B. ETS will seek, consider and, as appropriate, act on the 
views of those who sponsor, use or are affected by ETS 


programs and services. 


C. ETS will seek to obtain advice on its activities and 
policies from qualified men and women who are not 


employed or retained on a regular basis by ETS and 


who are drawn from appropriate professional disciplines, 
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major philosophies and points of view, different geographic 
regions, and the major ethnic groups within the relevant 


population. 


D. ETS will support the activities of professional associations 
with respect to developing and implementing professional 
standards or codes, making available the results of current 


work, and fostering peer review of its activities. 
Procedural Guidelines 


1. Information should be provided to sponsoring organizations with 
which ETS has contractual relationships in a form that permits 
evaluation of ETS services in terms of: 

a) quality; 
b) timeliness; 
c) costs; and 


d) responsiveness to legitimate comments or 
criticisms. 


2. Procedures should be established to facilitate communication 


with sponsors by: 


a) meeting at least annually to provide information 
and to receive comments on matters affecting 
the operations with which they are concerned; 


b) defining a mutually agreeable process to be 
used to transmit comments from sponsors or 
others and a time period within which the 
evaluation of comments will be completed 
and reports of actions to be taken by ETS 
can be expected. 


c) making available periodic opportunities for 
sponsors to express opinions, judgments and 
counsel concerning their activities or programs 
directly to ETS officers not normally responsible 
for such activities or programs. 
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Procedures should be established for making available technical 
and other information about ETS products and services to users 
so that they may evaluate the appropriate use of the product 


or service and communicate comments or criticisms to ETS. 


Procedures should be established to communicate with or provide 
information to persons who use or take ETS tests, who submit 
data for use by ETS or for distribution to others, or who 
participate in research and development projects conducted by 
ETS. This information should be communicated by ETS or the 
sponsor in such a way that these persons may understand their 


participation with respect to: 


a) the identity and scope of the sponsor's 
responsibility; 


b) the nature of the product, service or 
research by which they are affected, 


c) the way in which the product, service or 
research will likely be used by educational 
institutions or others; and 


d) the channels that have been established for 
addressing comments or criticisms to ETS 
or to the sponsor and response thereto. 


Organizational and program financial information should be 
recorded, processed and reported in accordance with generally 
accepted accounting principles and under appropriate 


safeguards to insure accuracy. 


An annual report that provides information about organizational 
activities and finances should be published by ETS on a regular 
basis and made available to any person on request. Program 


and project reports, including program financial information, 


should be made available in a manner consistent with contractual 


understandings. 
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7. Requests for information that is not included in an existing 
publication should be considered by the appropriate sponsor and 
by ETS. If its disclosure is consistent with applicable law, 
with ETS and sponsor policy, and with contractual obligations 
governing confidential or proprietary information, the 
information should be provided. If complying with a request 
for information results in a cost to ETS or a sponsor or 
affects the normal schedule of fulfilling ETS' responsibilities, 
ETS may provide the requested information in a reasonable 
period of time and at an appropriate price for the services 
rendered. Procedures should be established, as appropriate, 


to facilitate responses to these requests. 


8. Changes in federal statutes, regulations and case law that 
affect research and development, testing programs, or advisory 
and instructional services should be monitored to insure that 
ETS activities and operations are in compliance as relevant 
federal laws or rules change. Changes in other statutes, 
regulations and case law should be evaluated as appropriate for 


the same purpose. 


9. All proposed new ETS activities should be reviewed by counsel 
for compliance with applicable federal law and state law as 
appropriate. ETS officers and staff should direct the attention 


of legal counsel to matters that might affect ETS compliance. 


10. Advice should be sought, where appropriate, from men and 


women drawn from diverse backgrounds, interests and experience 


(e.g., appropriate professional disciplines, major philosophies 


and points of view, various geographic regions, and major ethnic, 
handicapped and other relevant subgroups of the population of 
interest) who are qualified to make a contribution to the 
direction and substance of ETS programs and who are not employed 


or retained on a regular basis by ETS. 


= Approved For Release 2006/12/27 : CIA-RDP83-001 56 R0003000 50.0 544 neers NSS 


EO e6§OC77Oo7O7 
Approved For Release 2006/12/27 : CIA-RDP83-00156R000300050051-4 


ll. 


12. 


13. 


14. 


-5- 


Individuals who become members of an ETS external advisory, 
review or evaluation committee should be informed about the 
results of the committee's work in a reasonable period of 


time. 


A reasonable accommodation should be made with respect to 

the professional responsibilities of the staff in order to 
permit staff members to attend professional meetings, to 
contribute to the development of professional standards or 
codes, to participate in and benefit from the dissemination of 
information on subjects of professional interest and to stay 
abreast of current concerns and accomplishments in related 


fields. 


ETS should have effective procedures for peer review whenever 
that will contribute substantially to the quality of ETS 


work. 


ETS should have effective and equitable procedures for handling 


questions of score authenticity arising in connection with the 


administration of tests. 


a 8 DD er || 
Approved For Release 2006/12/27 : CIA-RDP83-00156R000300050051-4 


ae aa Approved For Release 2006/12/27 : CIA-RDP83-00156R000300050051-4 


273 


CONFIDENTIALITY OF DATA 


Principle 


ETS recognizes the right of individuals and 
institutions to privacy with regard to 
information supplied by and about them that 
may be stored in data or research files held 
by ETS and the concomitant responsibility to 
safeguard information in its files from 
unauthorized disclosure. 


Policies 


A. ETS will ask individuals to provide information about them- 
selves only if it is potentially useful to those individuals, 
necessary to facilitate processing of data or serves the 
public interest in improving understanding of human performance. 
Insofar as possible, individuals should be informed of the 


purpose for which the information is requested. 


B. The right of individuals to privacy regarding information 
about them that may be stored in the data or research files 
held by ETS extends both to processed information, such 
as scores based on test-item responses, and the raw data on 


which the processed information is based. 


Cc. ETS will protect the confidentiality of data supplied by 
institutions or agencies about themselves,-and so identified, 
to the extent that such confidentiality does not conflict 


with ETS obligations to individuals. 


D. ETS will not collect or maintain in its data or research 


files any critical information that in its judgment cannot 


be protected adequately from improper disclosure. 
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E. ETS will encourage the organizations with which it works to 
adopt policies and procedures that adequately protect the 
confidentiality of the data transferred by ETS to those 


organizations. 
Procedural Guidelines 


1. Information about an individual, which has been identified as 
such, may not be released by ETS to organizations other than 
those for which the information was collected without the 
consent of that individual. A written exception may be made in 
the case of research studies during which the Committee on 
Prior Review of Research has determined that release of the 
data serves a public need, that there is no satisfactory and 
reasonable alternative way of obtaining the information, that 
the recipient researcher will use the data in appropriate ways 


and that there are adequate assurances of confidentiality. 


2. Information about an institution, which has been identified as 
such, may be released from ETS only in a manner consistent with 
a prior agreement or with the consent of the institution or 
with the approval of the cognizant ETS officer and representative 


of the appropriate sponsor (if any). 


3. An individual should be able, on payment of a reasonable 
fee, to authorize the disclosure of information about himself 
or herself from program data files held by ETS to any appropriate 
recipient, provided that such authorization is in writing and 
that disclosure is not inconsistent with other ETS or sponsor 
policies and does not violate the privacy of other individuals. 


Identification of the requester, through signature and data 


file number, or other appropriate method, should be required 


before any such information is released. 
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4. In an emergency and when it is to the benefit of the individual, 
an authorization by telegram or telephone for the release of 
personal data should be acceptable, provided that such authorization 
includes adequate identifying information and that such release 
is not inconsistent with other ETS or sponsor policies. By 
prior agreement with the individual, authorization by a designated 
agency or institution should also be acceptable. In such 
instances, the individual should be informed that the disclosure 


has taken place. 


5. If an individual is not competent because of illness or other 
considerations, information about that individual may be 
released from data files only with the consent of the individual's 


parent or legally appointed guardian. 


6. Unless the access to confidential data can be safeguarded, ETS 
should not participate in any time-sharing network, data bank, 
or other electronic data processing or storage system involving 


units outside ETS. 


7. On submission of appropriate identifying information and 
payment of a reasonable fee, an individual should be able to 
obtain information about himself or herself in ETS-held data 
files for the following purposes: to ascertain the accuracy of 
personal or biographical data and to request verification, 
within a reasonable period of time, of test scores or other 
processed information from tests, questionnaires, or school 
records, provided such release is consistent with sponsor 


policies. 


8. Procedures should be developed for systematically eliminat ing 


from data files information that is judged to be out of date 


and, hence, of minimal value. 
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9. Information from ETS~held program data files provided by 
individuals for a designated purpose should not be used or 
released for another purpose (such as a validity study or 
research project) without the individual's consent except 
when used or released in a form that cannot be identified 


with the individual. 


10. ETS should refuse to provide personally identifiable information 
except in accordance with these guidelines unless served with 
a subpoena or other court order. In that event, ETS should make 
appropriate efforts to quash or narrow the subpoena or order or 
to obtain a protective order to minimize the exposure of 


personally identifiable information. 


11. At the time information is collected and to whatever extent 
practical, programs should inform individuals of the conditions 
surrounding the release and confidentiality of the information 


about them. 


12. Individuals should be identified in ETS research files only by | 
code numbers. Information linking the code numbers to names 
should be kept in a secure location only as long as necessary 
for purposes such as follow-up studies or collating new data, 


after which the names should be destroyed. 


13. Every organization with which ETS works should be informed 
of the confidential nature of any data transferred by ETS to 
that organization or collected by the organization on behalf of 
ETS so that appropriate procedures can be employed by the 
recipient organization to protect the confidentiality of 


such data. 
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PRODUCT ACCURACY AND TIMELINESS 


Principle 


The accuracy of ETS' principal products and 
the timeliness with which they are made 
available are important parts of the 
responsibility ETS has undertaken with 
respect to its sponsors and the diverse 
public it serves. 


Policies 


A. ETS will establish standards of accuracy and timeliness with 


respect to each principal product. 


B. ETS will use quality controls that are adequate to assure that 


its standards of accuracy and timeliness are met. 


C. ETS will make realistic delivery commitments and reasonable 


efforts to meet those commitments. 


D. ETS will sacrifice the timeliness of the delivery of information 
if the desired accuracy of that information is substantially 


in question. 


E. ETS will seek to inform those adversely affected if, subsequent 
to its release, information has been found not to meet ETS 


standards of accuracy. 


F. ETS will seek to inform those adversely affected if there 


is a probability that there will be substantial departure from 


ETS standards of timeliness with respect to a principal product. 
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Procedural Guidelines 


1. Principal products should be identified and a standard of 
accuracy using units of measurement appropriate to the type of 


product should be established for each. 


2. When appropriate, quality control should include an adequate 
and independent recomputation and a visual reexamination of 
ETS-processed information based on an appropriate sample of 
cases sufficient to identify errors within the limits of the 


applicable standards of accuracy. 


3. When the computational nature of the information is such 


that it is impossible or impractical to determine the accuracy 


of the information by independent recomputation, staff members 


who are technically competent to do so should assess its 


"reasonableness" as a part of quality control. 


4. There should be a quality inspection of intermediate products 


when: 


a) the accuracy of variable information (e.g. 
parameter data, algorithms), verified by 
independent recomputation or assessment, 
influences the nature of an ETS process or 
computation and is critical to the process that 
generates an ETS principal product; or 


b) detection and correction of errors would 
facilitate meeting the delivery commitment 
on the principal product; or 


c) the nature of the principal product is such 
that it is impossible or impractical to 
determine the accuracy of the information 
by independent computation using the source 
data. 
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Quality control for principal products that do not contain 
processed information (such as bulletins of information or 
test books) should include inspection of a sample prior to 
release of the product. If the product is released from an 
outside vendor (e.g., outside publisher) or a sponsor's agent, 
quality control should include inspection of those components 
of the principal product that contain critical information on 


ETS-provided services. 


Quality control of information given in letter or telephone 


responses should include a periodic audit of a sample. 


Failure to meet standards of accuracy and timeliness should 


be reported to a designated ETS staff member for resolution. 


A principal product that does not meet established standards 

of accuracy should not be released until appropriate corrective 
action is taken unless release would be for the benefit of the 
score recipient and users and permission to release is given by 


the cognizant ETS officer. 


If an error is found in critical information already released 


by ETS, the correct information should be promptly distributed. 


Process control methods (e.g., a predefined schedule including 
a delivery date and contingency procedures for dealing with 

volume surge) should be established for the production of each 
principal product to help assure its delivery by the scheduled 


delivery date. 


If it is likely that there will be a substantial departure from 
ETS standards of timeliness with respect to a principal product, 


those who would be adversely affected should be so notified. 
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RESEARCH AND DEVELOPMENT 
Principle 


A continuing program of research and development 
conducted in compliance with professional standards 
with respect to quality and ethical procedures is 
necessary to maintain the high quality and social 
utility of ETS contributions to education. This 
includes basic inquiry to increase understanding 

of educational processes and human development; 
evaluative and applied research in response to 

the needs of the educational community; and research 
and development to improve ETS products and services. 
Publication of the results of significant ETS research 
is of benefit to ETS and the profession because it 
permits others to use, build upon or improve ETS 
work. 


Policies 


A. ETS will devote appropriate research efforts to improving 
education through the discovery and conceptual integration 
of new principles and understanding. This research will be 
aimed at extending knowledge of the learner and learning 
processes, of learning environments and educational treatments, 
of educational institutions and of the interacting factors 


that influence human development. 


B. ETS will devote appropriate research efforts to the improvement 
of the technical quality of ETS products and services. Among 
the important issues addressed by this research will be problems 
of test development, reliability, equating, validity, and 


meaningfulness of interpretation. 


Cc. ETS will devote appropriate research and development 
efforts to the identification of needs of the educational 


community and to the creation, improvement and evaluation 


of instruments, systems and programs of service that 


meet these needs. 
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ETS will conduct its research under appropriate procedures 
that protect the rights of privacy and confidentiality of 


human subjects or respondents. 


ETS will follow procedures to insure that ETS research is 

of high quality. Standards of quality in research refer 

to such matters as the identification of relevant data, the 
choice of suitable methods of collecting and analyzing data, 
the logic and objectivity of analysis and interpretation, 

the exploration of relationships between research problems and 


findings, on the one hand, and existing knowledge, theories and 


methodologies on the other, and the thoroughness and care of 


project planning and management. 


ETS will undertake research only if its potential benefits 
outweigh the inconveniences of or risks to the subjects or 


respondents who are involved. 


ETS will encourage the dissemination of full accounts of ETS 
research in the usual professional forums and will provide 
internal means by which the results of ETS research can be 


published. 
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Procedural Guidelines 


1. To maintain the quality of operational programs, ETS should 


engage in the following activities: 


a) study and research on the test development process, 
including systematic development and evaluation 
of new item types and approaches; 


b) studies to determine the sources of significant 
differential performance of sex, ethnic, handicapped, 
and other relevant subgroups on ETS tests; 


c) periodic evaluation of current approaches to aptitude 
and achievement measurement to determine fairness, 
validity and appropriateness for significant subgroups 
such as minorities and women; 


d) research related to reliability theory and practice, 
including methods of determining the reliability of 
classification decisions: 


e) study of the equating methods presently in use and 
development of improved methods as limitations in the 
applicability of the present methods are observed; and 


f) research to advance measurement techniques and selection 
and classification models relevant to fairness and validity. 


2. Research projects should be undertaken in such areas as 
learning and cognition, personality and social influence, 
teacher behavior and instructional processes, socialization 
and human development, and the economics and sociology of 
education as a means of improving educational policies and 


practices. 


3. Efforts should be made to develop instruments and programs 


of service in areas such as measurement, institutional 
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and program assessment and evaluation, instruction, 
guidance, financial aid, certification and licensing, and 
technology that would be of educational and social 


utility. 


4. Proposals for research to be conducted by ETS and involving 
human subjects or respondents should be considered by the 
Committee on Prior Review of Research, under its procedures 
for review, to verify that proper arrangements have been 
made for protection of the welfare and rights of human 


subjects. 


5. Researchers should not conduct research projects without the 


consent of subjects and respondents. In the case of young 
children, the consent of parents or a legal guardian, or of 


appropriate institutional representatives, should be obtained. 


6. Each research proposal should be reviewed by one or more 
persons who are competent in the field within which the 
proposal falls. They should be satisfied that professional 


standards of quality and ethical conduct are met. 


7. Identifiable data should be released from ETS to researchers 
other than those who originally conducted the research only 


when one of two conditions have been met: 


a) Consent to do so has been given by or on behalf of 


the subjects or respondents or by those who have 
given consent on their behalf; or 


b) the Committee on Prior Review of Research has 
determined that release of the data serves a public 
need, that there is no satisfactory and reasonable 
alternative way of obtaining the information, that the 
recipient researcher will use the data in appropriate 
ways and that there are adequate assurances of 
confidentiality. 
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8. After the data-collection phase of a research project has been 
completed, subjects should not be expected to provide additional 
data for a follow-up study unless such participation was part 
of their original agreement to serve as subjects, or their 
consent for follow-up is obtained or the follow-up study has 


been approved by the ETS Committee on Prior Review of Research. 


9. The results of measures of performance based on experimental 
situations or tests the interpretation of which is therefore 
tentative and whose applied use is not yet supportable should 
not be reported to subjects, or to the institutions providing 
the subjects, unless there is relatively little danger of 
misinterpretation or misuse of the information that would 
be harmful to those individuals or institutions or unless the 
use is part of a feasibility study or experimental condition. 
Stipulations regarding nonissuance of such reports should 


be made to participants in advance of the data collection. 


10. The results of each research project undertaken with respect 
to a particular ETS program or service should be available for 
dissemination unless a specific need to restrict publication 
to protect confidentiality or for other program purposes is 
identified prior to the beginning of the project and made 


known to the appropriate individuals. 


11. The contracts under which research is undertaken for agencies 
or institutions outside ETS should permit publication of 
the results of the research unless a specific need to protect 


the research results is identified prior to the beginning 


of the research and made known to the appropriate individuals. 
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TESTS AND MEASUREMENT 


This section which deals with ETS testing activities 
is divided into seven subsections that are devoted to test development, 
test administration, reliability, scale definition, equating, score 


interpretation, and validity. 


TECHNICAL QUALITY OF TESTS 


Principle 


High standards of quality and fairness in 
constructing, administering, reporting, 
interpreting and evaluating ETS tests are 
central to ETS' capability to function 
effectively as an educational service and 
research organization, 


Policies 


A. ETS will strive to develop tests in which the attributes 
measured, procedures followed, and criteria used will be 
unbiased with regard to a heterogeneous group of examinees 


and appropriate to the use for which the test is designed. 


B. ETS will establish standards for test~administration processes 
that minimize variations in test performance due to circumstances 


or conditions not relevant to the attributes being measured. 
Cc. ETS will establish for its tests a high degree of reliability 


(accuracy of measurement), consistent with the requirements and 


the purposes of the test. 
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D. ETS will develop scales for reporting scores in a rational 
fashion, consistent with the requirements and the purposes 


of the test. 


E. ETS will provide equating systems, when appropriate, for 
the perpetuation of scales for reporting scores at the highest 


level of precision practicable. 


F. ETS will make available to sponsors, institutional or agency 
users and examinees data for interpreting scores on ETS tests 


that foster appropriate use of those scores. 


G. Recognizing that test validation is a responsibility of both 
test users and test developers, ETS will encourage and assist 
test users in their validation efforts and will itself make 
available tests that are designed to meet professionally 
acceptable standards of validity provided the use of such tests 
is consistent with the primary purposes for which the tests 


were developed. 


H. ETS will adhere to appropriate professional standards such as 
those published in Standards of Educational and Psychological 
Tests and Principles for the Validation and Use of Personnel 


Selection Procedures. 
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Procedural Guidelines 


Section 1: Test Development 


Policy and substantive contributions to the test development 
process should be obtained from qualified men and women who are 

not on the full-time staff of ETS and who are drawn from 

diverse backgrounds and appropriate specialties within professional 
fields (e.g., various kinds of institutions and programs, 

relevant philosophies and points of view, and major ethnic, 


handicapped and other relevant subgroups of the population). 
Appropriate background information for use in the development 
of a test should be documented at appropriate stages in the 
development process and include: 

a) the purpose for which the test is intended to be used; 


b) the nature of the population that will take the test; 


c) the relevant procedural, financial or time constraints 
that will influence the available test development 
methods and their likely outcomes; 


d) for achievement tests, the kinds of curricula for 
which the test is designed; 


e) for job-related tests, the elements in training or 
employment that are related to performance on the job. 


For each test, specifications should be developed and reviewed 


by a process that provides information from the following 


perspectives: 
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a) content and skills--specifications should include the 
psychological, educational, or other domains to be 
sampled; the relative weight to be given to each domain; 
the appropriate level of proficiency to be required 


within each domain; a balance with respect to curricular 
differences. 


b) test and item format--specifications should include the 
item (question) types that are most clearly related to 


content or skills to be measured; the appropriate level 
of language or reading; requirements regarding clear and 
comprehensive directions and sample items or the need for 


a sample test; and whether free-response, multiple-choice 
or other machine scorable formats can be used. 


c) psychometric--specifications should include the level 
of difficulty of the test; the distribution of item 


difficulties (when pretested items are used); guidelines 
for evaluating the homogeneity among items within a test 
and the relationship between subtests or tests; equating 


requirements; number of items and time allotted. 


d) sensitivity--specifications for tests should require 
material reflecting the cultural background and 
contributions of women, minorities, and other subgroups; 


specifications should also require a balance of positive 
connotations if negative connotations are made in any 
references to these groups. 


4. Except for tests designed to measure rate of performance, the 
number of items in a test that has a specified time limit 
should be chosen so that time is not a decisive factor in 


performance, at least for the large majority of examinees. 


5. Subject matter and measurement specialists familiar with the 
purpose of the test and with the characteristics of the intended 
population should review the test items for accuracy, content 
appropriateness and the adequacy with which the items sample 


the domain. 
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6. The individual items in a test should meet appropriate technical 
standards such as those contained in the manuals for item writers 


used in the test development area. 


7. Individual test items and the test as a whole should be reviewed 
to eliminate language, symbols or content which are generally 
considered potentially offensive, inappropriate for major 
subgroups of the test-taking population or serving to perpetuate 
any negative attitude which may be conveyed toward these 
subgroups. No item in any test should include words, phrases 
or description that is generally regarded as biased, sexist or 


racist (e.g., demeaning modifiers and stereotypes). 


8. The items in a test should be reviewed by editorial specialists 


for clarity, accuracy, consistency, and, when appropriate, for 


conformity with standard editorial style. 


9. Tests should contain clear and complete directions. Enough 
sample problems should be provided in test-program publications 
so that the examinee can understand the nature of the task and 
the test-taking procedures. Where there is a need to provide a 
general orientation to testing, as when testing young children, 
practice tests--included either in descriptive material or at 


the time of test administration-~should be used. 


10. The typography, directions, and arrangement of items in the 
test booklet should facilitate the task of test-takers. When 
appropriate, tests should be made available to handicapped 
individuals such as sight-deficient candidates through tapes, 


readers or special printing. 
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Methods should be employed to evaluate the appropriateness 

of items before their operational use in a program or before 

the reporting of scores. Appropriate methods include pretesting, 
preliminary item analysis (using the first operational use of 
items as an opportunity to identify inadequate items) or 

careful review of the results of administering similar items to 
a similar population. In assessing the appropriateness of 

items before their operational use, efforts should be made to 
include representative samples of the operational test-taking 


population. 


The operational use of each test should be followed by systematic 
item analyses using appropriate criteria and by test analyses. 
These analyses should include reliability, intercorrelations of 


sections or parts, and speededness. 


Studies relating item performance to subgroups should be 
carried out for new or substantially revised tests when there 
are adequate data concerning sufficient samples of large 
subgroups whose education and experience may be different from 


the majority of examinees. 


The specifications for tests in ongoing programs should be 
reviewed for relevance and appropriateness before each new form 
is created. ETS staff and advisers should consider whether 
changes in the field, discipline or curricula require a revision 


of the specifications. 


When major changes are made in test specifications, consideration 
should be given to the implications of such changes for score 
comparability and whether it is necessary to change the test 


name or otherwise communicate to those who interpret test 


scores that comparisons with earlier tests may be inappropriate. 
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16. When test forms are used for a number of years in a program, 
they should be reviewed periodically for their appropriateness. 
The frequency of such review should be determined by the amount 
| of change occurring in the population of test-takers or the 


| subject matter domain. Test forms that are found to be outdated 
| 


should be revised or withdrawn from use. 
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Procedural Guidelines 
Section 2: Test Administration 
1. Information should be made available to prospective examinees 
and (in some programs, to parents or guardians as well) in 
advance of the test administration with respect to the following, 
as appropriate: 


a) the purpose of the test and what it measures; 


b) the nature of the test items (including samples of typical 
item types); 


c) the relevant instructions for taking the test, 
including instructions for guessing, changing 
answers, and strategy involving speed and accuracy 


in taking the test; 


d) identification requirements and the consequences of not 
having identification; 


e) the consequences of misconduct by the test-taker; 
£) background and experience relevant to test performance; 


g) the location of test centers, the test dates and 
special testing arrangements that can be made; 


h) the procedures for registering for the test and 
changing the centers; 


i) the structure of test fees and fee waivers; 


i) special arrangements available for administering tests 
to handicapped individuals; 
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k) the reporting of scores; 


1) procedures for canceling test scores by the candidate and 
reasons why ETS or the sponsor of the test might cancel 
scores; and 


m) the procedures for registering complaints. 


2. Program publications should be reviewed for language or 
descriptions generally regarded as biased and offensive. For 
example, the exclusive use of masculine pronouns should be 
avoided as should the implication that all persons in a given 
category (for instance, examinees, supervisors, counselors, or 
teachers) are either females or males (unless, of course, the 
category is logically restricted to members of a single sex). 
Illustrations, examples and practice items in test-information 


publications should represent males, females, minority and 


majority groups, and individuals in ways that indicate respect 


and awareness of valuable contributions. 


3. The facilities at which tests are administered should be 

| places that are convenient for the majority of examinees, 

| nonsegregated and comfortable. At least portions of those 
facilities should be accessible to and responsive to the needs 


of handicapped individuals. 


4. ETS should enlist test~center supervisors and staff with 
demonstrated sensitivity to the anticipated sex and ethnic 
composition of the examinee group, based on prior experience. 
When appropriate, persons affiliated with institutions attended 
by significant numbers of those examinees should be included. 
Minority-group supervisors and/or proctors should be employed, 
and test sites should be located in minority communities 


whenever appropriate and feasible. 
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5. Test-center supervisors and staff should be familiar with 
the procedures for administering a standardized test and 
should be provided with a description of the testing program, 
a description of the candidate population, and specific instruc- 
tions for administering the test. Instructions concern such 
subjects as the duties of test supervisors, associate supervisors, 
and proctors; the receipt, storage and return of test supplies; 
the admittance of examinees to the testing rooms; the distribution 
of test materials; procedures to be followed in administering 
tests to handicapped individuals; procedures to be followed in 
instances of suspected cheating; procedures to be followed in 
other cases of candidate misconduct; and procedures to be 


followed in case of emergency. 


6. Test performance can be affected by the psychological atmosphere 


of the testing center. Test supervisors should be informed of 


this and instructed to take measures to avoid an adverse situation. 
For example, test supervisors should be instructed, when it is 
appropriate and feasible, to have minority- as well as majority- 
group persons, women as well as men, read test directions and 

to recognize questions from examinees following an impartial 


procedure. 


7. ETS should provide the test center supervisor with directions 
to be read aloud to examinees before the test begins. These 
directions should include information relating to: procedures 
for marking answer sheets, timing of test sections, strategies 
for guessing, time and duration of test breaks and examinees' 
use of unauthorized aids. Test supervisors should check to see 
that examinees understand their task and the procedures to be 


followed. 
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Reasonable efforts should be made to eliminate opportunities 


for examinees to attain scores by fraudulent means by stipulating 


requirements for identification, assigning examinees to seats 


and requiring appropriate space between seats. 


Appropriate procedures should be applied after the test 
administration to identify scores of questionable authenticity, 


to resolve issues of authenticity and to provide for prompt 


reporting of questioned scores found to be authentic. 


A systematic program for observing test administrations 

should be conducted by trained ETS staff members or other 
qualified individuals: to review the testing procedures with 
the test supervisors, to insure appropriate testing conditions, 
to insure adequate maintenance of test security at the test 
centers and to relay questions and concerns from the field to 


the appropriate ETS office. 


Testing programs should have detailed procedures for investigating 
and resolving examinees’ complaints of irregular test administration 


or score report ing . 


Comments and suggestions should be solicited from supervisors 
by such means as the Supervisor's Comment Sheet and meetings 
of supervisors to provide ETS staff with information to 


improve future administrations. 


Supervisors should be required to record and report to ETS 
information on irregularities (such as mistiming, defective 
materials, power failures and cheating) so that ETS can 
evaluate the possible effect of such occurrences on examinees' 


test performance. 


An individual who has taken a test should be provided information 


that will be helpful in interpreting scores on that test. 
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Section 3: Test Reliability 


l. When test scores are reported to institutional or agency users 


or to individual examinees, information about the reliability 


of the test should be documented and should include: 


a) 


b) 


c) 


d) 


e) 


f) 


g) 


h) 


i) 


a reliability coefficient and an overall standard error of 
measurement (several indices may be provided if more than 

one method of assessing reliability has been used; alternate- 
form information should always be provided if available); 


standard errors of measurement for score regions if decisions 
about individuals are made in those score regions and if the 
overall regions and the overall standard error are judged 
inappropriate; 


the formula(s) used to estimate reliability and/or 
appropriate references; 


a justification of the method(s) used to assess reliability; 


a specification of the major sources of measurement error 
accounted for in the reliability analysis; 


a specification of the time interval between testings if 
alternate-form or test-retest reliability is used; 


the number of observations, the mean and standard deviation 
of the analysis sample (ranges or averages are acceptable 
in cases where the reliability information is derived 

from several samples); 


speededness data; and 


correlations of subscores within the same test or battery 
of which the test is a part. 
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2. If reporting any of the reliability information required under 
Guideline 1 is inappropriate, the reasons should be stated in 
appropriate program documents and, if possible, alternate 


information about consistency should be provided. 


3. Efforts should be made to provide reliability information in 
an appropriate form to the examinees to whom the scores are 


reported. 


4, The method(s) used for assessing reliability should: 


a) take into account the most common sources of error 
generally considered significant for test interpretation 
(e.g., guessing, instability over time, item and content 
variation, and rater inconsistency): and 


b) be appropriate to the nature of the test, in order not to 
seriously over- or underestimate reliability. 
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Section 4: Scale Definition 


1. Raw scores on a test or subtest (including percentages of 
questions answered correctly) should not be reported by ETS for 
individual examinees or in summary form for groups of examinees 


except under either of the following circumstances: 


a) when it is anticipated that only one edition of the test 
will be offered for use in the foreseeable future or it is 
demonstrated by appropriate empirical procedures that raw 
scores on all the editions to be compared are interchangeable; 
or when raw scores on that test edition will not be compared 
directly with raw scores on another test edition; or 


b) when reported in conjunction with a scaled score and in 
a context that supports appropriate interpretation, such 
| as when a copy of the test itself is available or when 
individual or group responses to individual items, depending 
on whether individual or group performance is being assessed, 
are available. 


2. If a test or test battery yields multiple scores for an individual 
and scaled scores are to be used directly (i.e., without 
reference to norms tables) in interpreting performance profiles, 
the scales should be normatively defined and each should be defined 


with respect to the same population. 


3. When different tests in a program are taken by different 
examinees whose scores are to be directly compared, the scales 


for the tests should take into account possible differences 


among the groups of examinees who take the various tests. 
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4. Established scales should not be redefined except under compelling 
circumstances. If a scale is to be substantially redefined, 
the numerical values should be changed substantially to minimize 
the possibility of confusion between test results expressed on 
the revised scale and results expressed on the original scale. 
An exception to this guideline may appropriately occur if the 
test in question is one of a set of tests for which a single 


range of numerical values (e.g., 20-80) is used and the scales 


for other tests in the set have not been redefined. 


5. Scale properties that affect score interpretation and use 
should be described in program publications available to the 


examinees and to institutional or agency users. 


6. Technical manuals and interpretive publications for institu- 
tional or agency score users and examinees should indicate, in 
language appropriate to the audience, whether a distributively 


based scale is intended to be normative or nonnormative. If it 


is intended to be normative, the group should be described. 


7. Whenever a normatively defined scale no longer conveys useful 
normative information, all published descriptions of the scale 


should be changed accordingly. 


8. Program publications should caution score recipients (users and 
examinees) that scores received on different tests that are 


reported on scales that are similar in appearance may not be 


equivalent. 
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Guidelines 9 through 14 apply only to scales established after 
guidelines 1-8 were published on August 1, 1977. 


9. If a scale is to be distributive, the choice between a normative 


and nonnormative distributive scale should take into account: 


a) the extent to which normative interpretation with 
reference to a particular population will be 
appropriate and useful for all examinees who take 
the test and for all purposes for which the scores 
are intended to be used; 


b) the probable time period during which the normative 
information conveyed by the scores will continue 
to be descriptively appropriate; and 


c) the feasibility of identifying and testing a suitable 
group of examinees on which to base a normative scale. 


10. The choice between a distributive and nondistributive scale 
should take into account the use for which the test was 


intended and to which the test is Likely to be put. 


ll. If a scale is to be defined with reference to standards of 
performance, the basis for establishing the standards should 


be determined empirically or rationally rather than arbitrarily. 
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The conventional grade- or age-equivalent score (the grade or 
age for which a particular score is the average) should not be 
used to establish the score scale for a test or system of 

tests. This type of score, as it typically has been derived, 
should be avoided altogether as a basis for reporting test 
performance. However, the grade (or age) for which a particular 
scaled score on a test is the average, referred to here as a 
"grade (age) level indicator" to distinguish it from the 
conventional grade-equivalent (age-equivalent) score, may be 
reported to help in score interpretation, if the practices 
customarily followed in deriving and presenting grade-equivalent 
(age-equivalent) scores are modified in accordance with ETS 
criteria that obviate the technical interpretive problems that 


grade-equivalent (age-equivalent) scores create. 


The choice of a scale should take into account the likelihood 


of confusion with other widely used scales. 


In establishing the number of distinct scale values to be 
reported, consideration should be given to the relative importance 
of the need to avoid erroneous distinctions among individuals 

(by reporting different scores for individuals whose true 

scores are the same) and the need to maintain distinctions 

that, on the average, will be correct (by reporting different 
scores for groups of individuals whose average true scores are 


different). 
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Procedural Guidelines 


Section 5: Equating 


Adequate equating should precede comparisons of the test 
performance of two or more individuals or groups on nonidentical 
items or sets of items such as test offerings in which successive, 


or alternate, forms are used interchangeably. 


Statistical methods selected for equating should be used only 
under circumstances that are consistent with the assumptions 


under which the methods have been developed. 


In regular and continuing testing programs that are available 
to users, integrated, long-range systems of equating the scores 
to all successive editions of the test should be used and 


described in technical publications. 


For those tests that are offered for institutional use (as 
distinguished from externally administered tests offered in 
testing programs) of which only a limited number of forms 

are available, equating of new forms should be based on specially 
designed studies in which examinees or groups of examinees are 
selected by an appropriate sampling procedure to take the 


alternate forms or alternate sequences of forms. 


When test forms are equated with the use of common (anchor) 
items, the psychological task of taking those items (represented, 
for example, by the directions, the context of the items and 


the speededness of the part of the test in which the items 


appear) should be the same for all examinees. 
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6. When the common items used for equating are not representative 
of the tests being equated, the groups of examinees used for 


equating should be as nearly as possible equivalent. 


7. In the continuing testing programs, statistical checks (e.g., 
check equating, special scale-stability studies) should 
be employed to permit regular assessment of the precision 


of the equating. 
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Procedural Guidelines 


Section 6: Score Interpretation 


Effective test use and meaningful score interpretation should 


be supported and augmented by: 


a) the development of appropriate test norms based on 
administering tests to samples from a defined population 
when there is a reasonable expectation that a large 
proportion of the schools or other units selected for the 
norms sample will agree to participate; or, 


b) a rationally developed system of interpretation shared with 
score recipients when score interpretation is not developed 
from normative data. 


Tests offered for sale and described by ETS as standardized 
tests (as distinguished from tests offered in testing programs) 
should have adequate norms or other information for use in 


interpreting test results. 


When test norms are developed by administering tests to samples 
from a defined population, the resulting norms should be 
representative of any relevant subgroup, including those 
defined by sex or ethnicity, in proportion to their frequency 
in the defined population. Such subgroups may be deliberately 
over-sampled for more precise estimation of the statistical 
characteristics of the population by procedures that take 
over-sampling into account. Data on the proportions in the 


sample and in the population, when available, should be reported 


in an appropriate technical publication. 
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4, The report of a special norms study should provide information 


on: 
a) the sampling design; 


b) the participation rate of institutions 
or individual respondents in the sample; 


c) characteristics of the participating 
institutions and individuals; 


d) weighting systems used in preparing 
norms; and 


e) estimates of sampling variability along 
with an acknowledgment, when necessary, 
that such estimates do not take into 
account biases arising from nonparticipation. 


5. When descriptive statistics based on program testing 
(as distinguished from norms based on special norms studies) 
are published, the following guidelines should be used: 


a) both table titles and descriptive 
material should make it clear that 
the statistics are based on examinees 
or participating institutions or 
other using agencies; 


b). the descriptive material should define the 
nature of the group by identifying the 
appropriateness of the sample and the factors 
that relate the background of the group to test 
performance, and by acknowledging explicitly 
that the sample is self-selected; 


c) when possible, reports should be prepared 
to show comparisons of data based on program 
examinees or institutional characteristics 


with relevant data on variables from other 
sources; 
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d) when information about interpretive data 
is prepared for different user groups, 
the presentation, whenever practicable, 
should be adapted to the needs and 
background of each group. | 


6. When norms are developed from program testing, the age, sex 
and ethnic composition of the program norms group should be 
described whenever such information about subgroup membership 


is available. 


7. In testing programs, descriptive statistics should be compiled 
periodically from a sample or entire population in order to 
monitor the participation and performance of males and females 
drawn from diverse backgrounds, interests and experience (e.g., 
major ethnic group, handicapped status and other relevant 


subgroups of the population of interest). 


8. If norms intended for use in the interpretation of individual 
scores are presented separately for males and females or for 
members of specific ethnic groups, the rationale should be 
carefully described. Separate norms may be justified for 
scores used primarily for guidance when access to the experiences 
needed to earn a high score is clearly related to subgroup 
membership and a more direct index of access is not available. 
The existence of score differences between subgroups does not 


in itself justify presentation of separate norms. 


9. Descriptive statistics prepared separately for subgroups of the 
relevant test-taking population but not intended for use in 


interpreting individual scores should not be presented in a way 


that encourages their use for such a purpose. 
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10. Institutional or agency users and examinees should be informed 
of the standard error of measurement of a score, and test 
interpretation materials should point out the limitations of 
test scores and encourage score users to take into account the 


possible scores a test taker might achieve on retesting. 


11. Statistical data used in score interpretation should be revised 
annually except when less frequent revision is judged to be 
appropriate as, for example, when norms are based on special 


studies. A statement of the period in which the data were 


collected should be included in any publication that presents 


the data. 


13. Institutional or agency score recipients should be provided 
with interpretive materials designed to be helpful for using 
scores in conjunction with other information, setting cutting 
scores where appropriate, interpreting the scores for special 
subgroups (e.g., ethnic minorities, males, females, and 


handicapped students), conducting local normative studies, 


and developing local interpretative materials. 
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Procedural Guidelines 


Section 7: Test Validity 


1. ETS should provide evidence of the validity of its tests in 
relation to the principal purposes or intended uses of the 


tests. One or more of the following may be applicable: 


a) when test scores are to be interpreted in terms of degree 
of mastery of the knowledge, skills, or abilities of a 
domain represented by the test, content validation evidence 
should be provided. 


b) when test scores are to be interpreted in terms of the 
prediction of future behavior, criterion-related validation 
evidence should be provided. 


c) when test scores are to be interpreted as a measure of a 
theoretical construct, construct validation evidence should 
be provided. 


2. Evidence of content validity should be based (a) on a careful 
determination and analysis of the domain(s) of interest and of 
the relative importance of topics within the domain, and (b) on 
a demonstration that the test is an appropriate sample of the 


knowledge or behavior in the domain(s). A report on evidence | 


of content validity should present descriptions of the procedures 
employed in the study, including the number and qualifications 
of experts involved in the analysis of the domain or evaluation 


of the relevance and appropriateness of the test. 


3. Construct validation should be based on: rational and empirical 
analyses of processes underlying performance on the test 
in question including, where appropriate, noncognitive as well 
as cognitive functions. Empirical evidence relevant to the 
analyses should include results of investigations of the degree 
to which test scores are related or unrelated to other variables 


in ways implied by intended interpretations. 
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4. Criterion-related validation should be used only when technically 
sound and relevant criteria are available or can be developed 
and when other conditions affecting feasibility warrant the 


study. 


a) Criterion-related validation should involve as many 
performance variables as necessary to permit evaluation of 
the effectiveness of test scores for predicting the types 
of behavior they are intended to measure. 


b) Criterion-related validation should not combine variables 
to form a single criterion measure unless such a procedure 
is justified by logical considerations or empirical evidence 
or the practical requirements of the intended use of the 
results. 


c) Criterion data should be collected in a way that permits an 
assessment of the reliability of each criterion variable, 
but with the understanding that there may be several 
sources of irrelevant variation, (sampling of criterion 
content, source of criterion ratings or data, and so 
forth). 


5. Interpretations of correlations between test scores and criterion 
variables should take into account such factors as sample size, 
criterion reliability, possible restriction in the range of 
scores obtained in the validity study sample, and other contextual 


factors. 


6. The method(s) by which any validation is accomplished should be 
fully documented; such documentation should include appropriate 
details such as the nature and reliability of the criteria, a 
description of the subjects used, the materials surveyed and the 
qualifications of the experts who made judgments regarding the 


appropriateness and importance of test content. 
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Where adequate methods are employed to insure equivalence of 
scores on alternate forms, it is not necessary that each new 
form be validated. New validation studies should be made if 
revised tests have substantial changes, such as different item 


types, or if they sample a revised performance domain. 


When appropriate and feasible, the validity of a test should be 
investigated separately for subsamples of the test-taking 


population. 


When a name of a test is established, it should not imply more 


than is justified by evidence of validity. 


Information should be made available to institutional and 
agency users that would be of assistance to them in planning 


and conducting local validity studies. 
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TEST USE 


Principle 


Proper and fair use of ETS tests is essential 
to the social utility and professional 
acceptance of ETS work. 


Policies 


A. ETS will set forth clearly to sponsors, institutional or agency 
users, and examinees the principles of proper use of tests and 


interpretation of test results. 


B. ETS will establish procedures by which fair and appropriate 
test use can be promoted and misuse can be discouraged or 


eliminated. 


Procedural Guidelines 
1. Program publications should: 


a) describe appropriate usés and caution against potential 
misuses of program tests; 


b) explain clearly that -test scores reflect past opportunity 
to learn and discourage test interpretations that go 
beyond reasonable inferences from test performance; 


c) emphasize that an individual's test score should be 
interpreted in the context of other information about 
him or her; 


d) provide appropriate information about test content, 
difficulty, and purpose to help the institutional or 
agency user select instruments that meet the measurement 
requirements of the situation and avoid selecting, 
requiring or using inappropriate tests; 
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e) invite institutional or agency users to consult with the 
program sponsor and/or ETS about their current or intended 
uses of ETS-developed tests and identify the offices 
to be contacted for this purpose; 


f) summarize results of research relevant to the use of the 
test or cite references in which such results are reported; 


g) describe adequately and clearly scale properties 
that affect score interpretation and use; 


h) advise institutional or agency users that decisions about 
the application of single or multiple prediction equations, 
based on distinguishing characteristics such as sex, 
ethnic group or curricular emphasis or training, should be 
preceded by careful examination of social, educational and 
psychometric factors; 


i) advise institutional or agency users that if examinee 
grouping based on test scores is practiced, provision 
should be made for frequent review of group assignments to 
determine actual performance; 


3) stress that pass-fail or cut-off scores established for 
such purposes as admission, credit, or certification, 
should be used as a basis for decision making only 
if the institutional or agency user has a carefully 
developed rationale, justification, or explanation 
of the cutting score that is adopted; and 


k) encourage institutional or agency users to reexamine 
cut-off score policies periodically to minimize or eliminate 
possible disproportionate exclusion of members of any group 
such as men and women drawn from diverse backgrounds (e.g., 
major ethnic, handicapped and other subgroups of the population 
of interest) in the face of other evidence that would predict 
their success or indicate their competence. 


2. Special (nonprogram) publications should be developed and 
disseminated by ETS to promote fair use of tests and discourage 


misuse of tests. 
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Complaints or information about questionable interpretation or 
use of reported scores should be investigated by means of 
procedures designed for detecting misuse. Such procedures 
should be documented, and records should be kept of such 


complaints and their disposition. 


In cases where a clear misuse is brought to its attention, 
ETS should inform the sponsor and the institutional or agency 
user of ETS' opinion as to the misuse and seek voluntary 
correction of the misuse. If reasonable efforts to seek 
voluntary correction are not successful, ETS, in conjunction 
with the sponsor, should take steps to determine whether to 


continue supplying tests or reporting scores to the institutional 


or agency user. 
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TECHNICAL ASSISTANCE, ADVICE, AND INSTRUCTION 


Principle 


ETS is dedicated not only to providing measurement 
programs and conducting research but also to promoting 
increased understanding of measurement and test use. 


Policies 


ETS will develop and offer instructional programs in the areas 
of measurement, evaluation, and related research through such 
forms as publications, seminars, in-service training, intensive 
residence courses, workshops, internships and conferences. ETS 
may undertake these activities independently or in cooperation 
with other agencies, professional groups or educational 


institutions. 


ETS will provide advice and information on measurement-related 
issues and about ETS programs, research and services. In this 
activity, ETS will work, where feasible, in collaboration with 
other professional organizations that show a concern about 


measurement. 


ETS will respond promptly to requests for advice, instruction 
and technical assistance related both to programs and services 
offered by ETS and to the related areas of educational measurement , 


evaluation and research. 


ETS will conform to high standards of accuracy and professionalism 


in its advisory, instructional and technical assistance 


activities, 
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ETS will provide advice, instruction and technical assistance 

to clients from the private and public sectors and from foreign and 
domestic government agencies to the extent that such services 

are consistent with ETS areas of expertise, meet accepted professional 
and ethical standards, and reflect an understanding of and respect for 


cultural differences. 


ETS will endeavor to promote increased understanding of the 
purposes and procedures of testing among professional groups 
and in the public sector; ETS will make this effort both 
independently and in cooperation with other organizations that 


share this responsibility. 


Procedural Guidelines 


ETS's offices should offer advice, instruction and technical 
assistance; the staffing for such services should be determined 


by the nature of the services and the expertise required. 


The special requirements of audiences with varying needs, 
interests, cultural backgrounds and levels of knowledge 
should be considered when ETS provides technical assistance, 


advice, or instruction. 


New developments in research or testing should be considered 


when technical assistance, advice and instruction are offered. 


Technical assistance, advice and instruction offered to 
institutions or agencies should include guidance on how to use 
other information about examinees (such as previous academic 


performance, English as a second language, and family or 


cultural background factors) in conjunction with test scores. 
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5. Comprehensive collections of reference materials relating to 
tests, measurement, evaluation and related research should be 
developed, maintained and made available to all ETS staff 
members and, when appropriate, to professional groups and 


individuals outside the organization. 
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GLOSSARY OF TERMS 


Accuracy: The extent to which a principal product conforms to 
its specifications or correctly reflects the source data within 
the specified limits of reliability. 


Client: (See Sponsor) 


Consent: Permission granted by an individual or that individual's 
parent or guardian to the use or release of data held by ETS; such 
permission granted upon receipt of a reasonable explanation of the 
purpose of the use or release and a reasonable explanation of the 
manner in which the results will be reported. 


Critical Information: Information that will be used to draw important 
inferences (a) about the sponsor, ETS-appointed external committees, 
institutional or agency user, examinee, subject or respondent, 

or (b) by the sponsor, institutional or agency user, examinee, 

subject or respondent and which, if incorrect, could be harmful. 


Distributive Scale: A scale that is defined to yield either a 
specified score distribution or a specified mean and standard 
deviation for a particular group of examinees. 


ETS Board of Trustees: The ETS Board of Trustees is the governing 

body of ETS. There are 16 trustees. Thirteen are elected for 
four-year terms. New members of the Board are elected by current 

trustees. Some are chosen from nominees proposed by the American 
Council on Education and the College Entrance Examination Board, 
two of the founding organizations of ETS. The presidents of the 
American Council on Education, the College Entrance Examination 
Board and ETS also serve as trustees. 


ETS-held Program Data Files: Information about individuals and 
institutions held by ETS and derived from ETS-provided services 
of collection, processing, storage, retrieval and dissemination. 


ETS-held Research Files: Information held by ETS and generated 
through ETS-conducted research intended to result in the develop- 
ment of new or improved techniques and materials for application 
in such areas as classroom instruction, evaluation of progress 
toward educational goals, counseling of students, and decision- 
making of school administrators. 


Examinee: An individual who takes a test, developed and or administered 
by ETS. 


Institutional or Agency User: An organizational recipient of 
ETS-processed or produced information. 


Intermediate Product: Materials that are not released externally, 
but that are necessary to the production of the principal product. 
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Glossary (continued) 


Nondistributive Scale: A scale that is defined without reference 
to the observed test performance of a particular group. 


Nonnormative Scale: A scale that is based on the performance 
of any conveniently available subgroups of examinees for whom 
the test is appropriate. A score on a nonnormative scale is 
not intended to convey information about an examinee's standing 
in relation to a defined population. 


Normative Scale: A scale that is based on the test performance 

of a sample of examinees, selected as prescribed by a specified 
design, from a clearly defined population. A score on a normative 
scale is intended to convey useful information about the performance 
of a particular examinee in relation to the performance of that 
population. 


Principal Product: ETS-produced or processed materials (e.g., 

annual reports, performance data, score reports and admissions 
tickets) that are released or transmitted to a sponsor, ETS-appointed 
external committee, institutional or agency user, examinee, subject 
or respondent, pursuant to a contract or published commitment. 
Standards with respect to accuracy and timeliness are applicable to 
principal products. 4 


Principles for the Validation and Use of Personnel Selection Procedures, 
Division of Industrial-Organizational Psychology, American Psychological 
Association. Dayton, Ohio: The Industrial-Organizational Psychologist, 
1975. 


Respondent: An individual who provides data to a research project 
in a manner and for a purpose different from either examinees or 
subjects. 


Sponsor: Educational, professional or occupational associations, 
federal, state or local agencies, public or private foundations 
which contract with ETS for its services. This category includes 
their governing boards, membership, and appointed committees or 
staff. 


Standards for Educational and Psychological Tests, American 


Psychological Association (APA), American Educational Research 
Association, and National Council on Measurement in Education. 
Washington, D.C.: APA, 1974. 


Subgroup: A part of the larger population which is definable 
according to various criteria as appropriate, e.g., by (a) sex, 
(b) race or ethnic origin, (c) training or formal preparation, 
(d) geographic location, (e) income level, (f£) handicap, (g) age. 
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Glossary (continued) 


Subject: an individual who participates in an ETS laboratory 
or experimental research project. 


Testing Program: A set of arrangements under which examinees 

are scheduled to take a test under standardized conditions, the 
tests are supplied with instructions for giving and taking then, 
and arrangements are made for scoring the tests, reporting the 
scores, and providing interpretative information as part of a 
comprehensive ongoing service. A program is characterized by 

its continuing character and by the inclusiveness of the services 
provided. 


Timeliness: The degree to which a principal product is released 
or delivered to its recipient within a predefined schedule. 


